Extracting Pre/Post Correlations in Repeated Measures Designs for use

in Meta-Analysis

Matthew Jané (The University of Connecticut)
Tylor Harlow (The University of Connecticut)
Saachi Shah (The University of Connecticut)
Ella Veiner (The University of Connecticut)
Enrico Khu (The University of Connecticut)
Tylor Harlow (The University of Connecticut)

The use of repeated measures designs is prevelent across a wide variety of scientific disciplines, and has become a frequent subject of meta-analytic syntheses. To calculate effect-sizes for repeated measures designs, an essential parameter is the correlation between pre and post-test scores. Despite this, pre/post correlations are frequently unreported in primary studies. As a result of the lack of awareness of alternative methods for calculating pre/post correlations, meta-analysts resort to the use of ballpark approximations in order to account for unavailable pre/post correlations. As innacurate pre/post correlations lead to innacurate results, this highlights the need for a systematic guide for calculating and accurately approximating pre/post correlations, hence the purpose of this paper. This paper will contain the necessary equations and code for various scenarios where different information is available

Introduction

Meta-analyses synthesizing repeated-measures designs are popular for drawing inferences about within-person effects over time. While these meta-analyses are capable of providing insight into within-participant effects, when conducted improperly they can lead to biased results (Cuijpers et al. 2017). Specifically, repeated-measures standardized mean differences (rmSMDs) rely on pre/post correlations in their calculation (Table 1). However, frequently these pre/post correlations are unavailable, leading meta-analysts to make inaccurate calculations. Some authors have gone so far as to recommend against the use of rmSMDs altogether (Cuijpers et al. 2017, 364):

We conclude that pre-post SMDs should be avoided in meta-analyses as using them probably results in biased outcomes.

Despite this cautionary stance, we advocate for a more nuanced approach. Numerous statistical methods are available to directly calculate pre/post correlations from alternative statistics, mitigating the risk of bias. Therefore, we believe that dismissing rmSMDs entirely may be an overly hasty response to the challenge at hand. Instead, we think that this dilemma stems largely from the lack of clarity on when and how to use alternative statistics when calculating pre/post correlations for rmSMDs. We look to address this dilemma by establishing a concise guide for calculating pre/post correlations depending on the available statistics (see Figure 1).

Defining the pre/post correlation

As we will see in the next section, pre/post correlations are present in the equations for rmSMDs. The pre/post correlation is a measure of the stability of individual differences from pre-test to post-test. The population pre/post correlation can be defined as the covariance between pre- and post-test scores divided by the product of the standard deviations of pre-test and post-test scores such that,

\[ \rho_{12} = \frac{\sigma_{12}}{\sigma_1\sigma_2} \]

Within a sample, we can compute Pearson’s product-moment estimator (Pearson and Filon 1897) which replaces the population values with the sample covariance (\(S_{12}\)) and standard deviations (\(S_1\) and \(S_{2}\)),

\[ r_{12} = \frac{S_{12}}{S_1 S_2} \]

Repeated Measures Standardized Mean Differences

Repeated Measures Standardized Mean Differences (rmSMDs) are used to quantify the change in an outcome from pre-test (i.e., time 1) to post-test (i.e., time 2). Estimators of rmSMD can be calculated in multiple different ways (see Table 1). All rmSMDs follow the same structure, that is, the difference in the post-test mean (\(\overline{Y}_2\)) and pre-test mean (\(\overline{Y}_1\)) divided by some standardizer (\(S_*\)),

\[ d_* = \frac{\overline{Y}_2 - \overline{Y}_1}{S_*} \]

The standardizer will be some kind of standard deviation (e.g., standard deviation of change scores). Of the four types of rmSMDs described in Table 1, While one may want to choose an rmSMD standardizer that does not contain the pre/post correlation coefficient (see Table 1), the sampling variance of these estimators still require the pre/post correlation. Therefore, proper calculation of the rmSMD always necessitates the calculation of a pre/post correlation.

Estimator	Standardizer (\(S_*\))	Variance
Change score \(d_z\)	\[\sqrt{S^2_1 + S^2_2 + 2r_{12}S_1S_2}\]	\[ \frac{1}{n} + \frac{d^2_z}{2n}\]
Repeated Measures \(d_{rm}\)	\[\sqrt{\frac{S^2_1 + S^2_2 + 2r_{12}S_1S_2}{2(1-r_{12})}}\]	\[ \left(\frac{1}{n} + \frac{d^2_{rm}}{2n}\right) \times 2(1-r_{12})\]
Average Variance \(d_{av}\)	\[\sqrt{\frac{S_1^2 + S_2^2}{2}}\]	\[\frac{2(S^2_1 + S^2_2 + 2r_{12}S_1S_2)}{n(S^2_1 + S^2_2)}\]
Pre-test score \(d_b\)	\[S_1\]	\[\frac{2(1-r_{12})}{n}+\frac{d^2_b}{2n}\]

Table 1: Equations for the standardizer and sampling variance for different types of rmSMDs obtained from Jané et al. (2024). Note \(S_1\) = pre-test standard deviation, \(S_2\) = post-test standard deviation, \(r_{12}\) = pre/post correlation, \(n\) = sample size.

Obtaining Pre/Post Correlations

Here we present a systematic procedure for calculating pre/post correlations depending on what information is available to the meta-analyst. Using a decision tree-like procedure (see Figure 1), we can can prioritize exact methods of obtaining pre/post correlations as opposed to approximations, which become a last resort if all other information is unavailable. Each step in the diagram displayed in Figure 1 will have its own section dedicated to overviewing the calculation of pre/post correlations using the information available for that given step.

In [2]:

graph TB
    A((Pre/Post\n Correlation))
    S0(Scenario 0:\n Is the pre/post correlation or raw data available?) --No--> S1(Scenario 1:\n Is the change score standard\n deviation available?)
    S1 --No--> S2(Scenario 2:\n Is the SMD of change scores available?)
    S2 --No--> S3(Scenario 3:\n Is the paired t-statistic available?)
    S3 --No--> S4(Scenario 4:\n Is the p-value from a\n paired t-test available?)
    S4 --No--> S5(Scenario 5:\n Is a figure available?)
    S5 --No--> S6(Scenario 6:\n Is the standard deviation\n response ratios available?)
    S6 --No--> S7(Scenario 7:\n Are there other similar studies\n with pre/post correlations?)
    S7 --No--> S8(Scenario 8:\n No usable information\n available)
    S8 --Rule of Thumb--> A
    S7 --Reasonable\n Approximation--> A
    S6 --Close\n Approximation--> A
    S5 --Close\n Approximation--> A
    S4 --Exact\n Calculation--> A
    S3 --Exact\n Calculation--> A
    S2 --Exact\n Calculation--> A
    S1 --Exact\n Calculation--> A
    S0 --Yes--> A

Pre/post correlation calculation scenarios

Obtaining pre- and post-test means and standard deviations

Before we get into the following scenarios, In most of the scenarios, estimates of the pre- and post-test means standard deviation will be required. Sometimes these are not directly reported in primary studies, a couple of situations may occur: 1) standard errors of the mean are reported instead of standard deviations, 2) confidence intervals of the mean are reported instead of standard deviations, 3) boxplots or five-point summaries are reported instead of means and standard deviations, 4) inter-quartile range is reported instead of standard deviations 5) the min-max range is reported instead of standard deviations.

Standard errors to standard deviations

Standard deviations can be calculated from the standard error (\(se\)) by multiplying the standard error by the square root of the sample size (\(n\)),

\[ S = se\times\sqrt{n} \qquad(1)\]

Where \(se\) is the standard errors of the pre-test and post-test respectively. It is important to note that some primary studies may misreport standard deviations as standard errors and vice versa, so it is important to cross-check with other statistics (e.g., t-statistics).

Confidence intervals to standard deviations

The confidence interval of the sample mean can be converted to a standard deviation by taking the range of the confidence interval (\(CI_U-CI_L\), where the subscripts \(U\) and \(L\) denote the upper and lower bound) and dividing and converting it to a standard error by dividing it by 3.92, then multiplying it by the square root of the sample size,

\[ S = \frac{CI_{U} - CI_{L}}{3.92} \times \sqrt{n} \qquad(2)\]

Boxplots and five-number summaries to means and standard deviations

If studies report distributions as a five-number summary (often displayed as a boxplot) with a minimum (\(Y_{\min}\)), 25th percentile (\(Y_{25}\)), median (\(Y_{50}\); i.e., 50th percentile), 75th percentile (\(Y_{75}\)), and maximum (\(Y_{\max}\)) we can use these values to approximate the mean and standard deviation. To approximate the mean, we can use the following formula (Luo et al. 2018, eq. 15)

\[\begin{aligned} \overline{Y} \approx \left(\frac{2.2}{2.2+n^{.75}}\right) \frac{Y_\text{min} + Y_\text{max} }{2}\\ + \left(.7-\frac{.72}{n^{.55}}\right) \frac{Y_{75} + Y_{25}}{2}\\ + \left(.3+\frac{.72}{n^{.55}}-\frac{2.2}{2.2+n^{.75}}\right) Y_{50} \end{aligned} \qquad(3)\]

and to approximate the standard deviation we can use the following formula (Shi et al. 2020, eq. 15),

\[\begin{aligned} S \approx \left(\frac{1}{1+.07 n^{.6}}\right)\frac{Y_\text{max} - Y_\text{min}}{2\Phi^{-1}\left[\frac{n-.375}{n+.25}\right]} \\+ \left(\frac{.07n^{.6}}{1+.07 n^{.6}}\right) \frac{Y_{75} - Y_{25}}{2\Phi^{-1}\left[\frac{.75n-.125}{n+.25}\right]} \end{aligned} \qquad(4)\]

Where \(\Phi^{-1}[\cdot]\) is the inverse of the cumulative distribution function of the standard normal distribution (in R, the qnorm() function is equivalent). Although these formulas are quite complex, the conv.fivenum() function in the metafor R package (Viechtbauer 2010) can do these calculations easily. An example in R is shown below:

In [4]:

library(metafor)

Loading required package: MatrixLoading required package: metadatLoading required package: numDeriv
Loading the 'metafor' package (version 4.2-0). For an
introduction to the package please type: help(metafor)

      mean        sd
1 3.756389 0.8248168

Inter-quartile interval and median to means and standard deviations

If the full five-number summary is not reported and instead the primary study only reports the inter-quartile interval (i.e., \([Y_{25}\),\(Y_{75}]\)) then we can calculate a different approximation for the mean and standard deviation. The following formula can be used to approximate the standard deviation (Wan et al. 2014, eq. 16),

\[ S \approx \frac{Y_{75} - Y_{25}}{2\Phi^{-1}\left[\frac{.75n-.125}{n+.25}\right]} \qquad(5)\]

If the median is reported instead of the mean, we can approximate the mean with the following formula (Luo et al. 2018, eq. 11),

\[ \overline{Y} \approx \left(.7 + \frac{.39}{n}\right) \frac{Y_{75} + Y_{25}}{2} + \left(.3 - \frac{.39}{n}\right)Y_{50} \qquad(6)\]

Using the same function as before (conv.fivenum()), we can calculate the mean and standard deviation only from the sample size and the 25th, 50th (median), and 75th percentiles:

In [6]:

conv.fivenum(q1 = 3.1,     # 25th percentile
             median = 3.7, # median (50th percentile)
             q3 = 4.5,     # 75th percentile
             n = 50)       # sample size

     mean       sd
1 3.77078 1.068626

Min-max interval to means and standard deviations

If the full five-number summary is not reported and instead the primary study only reports the min-max interval (i.e., \([Y_\min\),\(Y_\max]\)) then we can calculate a different approximation for the mean and standard deviation. The following formula can be used to approximate the standard deviation (Wan et al. 2014, eq. 7),

\[ S \approx \frac{Y_\text{max} - Y_\text{min}}{2\Phi^{-1}\left[\frac{n-.375}{n+.25}\right]} \qquad(7)\]

If the median is reported instead of the mean, we can approximate the mean with the following formula (Luo et al. 2018, eq. 7),

\[ \overline{Y} \approx \left(\frac{4}{4+n^{.75}}\right) \frac{Y_\text{max} + Y_\text{min}}{2} + \left(\frac{n^{.75}}{4+n^{.75}}\right)Y_{50} \qquad(8)\]

Using the same R function as before (i.e., conv.fivenum), we can calculate the mean and standard deviation only from the sample size, minimum, maximum, and median in R:

In [8]:

conv.fivenum(min = 2.2,     # 25th percentile
             median = 3.7,  # median (50th percentile)
             max = 5.1,     # 75th percentile
             n = 50)        # sample size

      mean        sd
1 3.691229 0.6463609

Scenario 0: The pre/post correlation or raw data is reported.

This is the ideal scenario where the pre/post Pearson correlation is reported in the primary study or the raw data is available. If the correlation is not reported, but the raw data is available then we will have to calculate the pre/post correlation ourselves. This can be done easily in base R using the cor() function. If neither the raw data or Pearson correlation is available, contact the authors of the primary study to obtain the raw data.

If the pre/post correlation is reported as a Spearman rank-order correlation instead of the Pearson correlation, we can use the Spearman rank-order correlation (\(r^R_{12}\)) to approximate the Pearson correlation assuming the underlying distribution is a bivariate normal (Rupinski and Dunlap 1996, eq. 2),

\[ r_{12} \approx 2\sin^{-1}\left(\frac{\pi \times r^{R}_{12}}{6}\right) \qquad(9)\]

However, if the pre/post correlation is reported as a Kendall’s \(\tau\) coefficient then the correlation tends to be under-estimated. Therefore, we can use Kendall’s (1962) formula for converting to a Pearson correlation (assuming an underlying bivariate normal distribution),

\[ r_{12} \approx \sin^{-1}\left(\frac{\pi\times \tau_{12}}{2}\right) \qquad(10)\]

Note that Equation 9 and Equation 10 are merely approximations and therefore it is recommended to move on to scenarios that allow for an exact calculation of the pre/post correlation before using these formulas.

Scenario 1:

Acknowledgments

The authors express our sincere gratitude for the invaluable insights provided by X. Through their generosity and expertise, this study has been significantly enhanced and safeguarded against numerous potential errors. Any remaining errors are solely our responsibility.

Cuijpers, P., E. Weitz, I. A. Cristea, and J. Twisk. 2017. “Pre-Post Effect Sizes Should Be Avoided in Meta-Analyses.” Epidemiology and Psychiatric Sciences 26 (4): 364–68. https://doi.org/10.1017/S2045796016000809.

Jané, Matthew B., Qinyu Xiao, Siu Kit Yeung, Mattan Ben Shachar, Aaron Caldwell, Denis Cousineau, Daniel Dunleavy, et al. 2024. Guide to Effect Sizes and Confidence Intervals. https://doi.org/10.17605/OSF.IO/D8C4G.

Kendall, Maurice George. 1962. Rank Correlation Methods. Griffin.

Luo, Dehui, Xiang Wan, Jiming Liu, and Tiejun Tong. 2018. “Optimally Estimating the Sample Mean from the Sample Size, Median, Mid-Range, and/or Mid-Quartile Range.” Statistical Methods in Medical Research 27 (6): 1785–805. https://doi.org/10.1177/0962280216669183.

Pearson, Karl, and L. N. G. Filon. 1897. “Mathematical Contributions to the Theory of Evolution. IV. On the Probable Errors of Frequency Constants and on the Influence of Random Selection on Variation and Correlation. [Abstract].” Proceedings of the Royal Society of London 62: 173–76. https://www.jstor.org/stable/115709.

Rupinski, Melvin T., and William P. Dunlap. 1996. “Approximating Pearson Product-Moment Correlations from Kendall’s Tau and Spearman’s Rho.” Educational and Psychological Measurement 56 (3): 419–29. https://doi.org/10.1177/0013164496056003004.

Shi, Jiandong, Dehui Luo, Hong Weng, Xian-Tao Zeng, Lu Lin, Haitao Chu, and Tiejun Tong. 2020. “Optimally Estimating the Sample Standard Deviation from the Five-Number Summary.” Research Synthesis Methods 11 (5): 641–54. https://doi.org/10.1002/jrsm.1429.

Viechtbauer, Wolfgang. 2010. “Conducting Meta-Analyses in R with the metafor Package.” Journal of Statistical Software 36 (3): 1–48. https://doi.org/10.18637/jss.v036.i03.

Wan, Xiang, Wenqian Wang, Jiming Liu, and Tiejun Tong. 2014. “Estimating the Sample Mean and Standard Deviation from the Sample Size, Median, Range and/or Interquartile Range.” BMC Medical Research Methodology 14 (1): 135. https://doi.org/10.1186/1471-2288-14-135.